(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(43) Date of publication: 

08.05.2002 Bulletin 2002/19 



(H) EP 1 204 277 A2 

EUROPEAN PATENT APPLICATION 

(51) IntCI 7 : H04N7/24 



(21) Application number: 01308791.1 

(22) Date of filing: 16.10.2001 



(84) Designated Contracting States: 


(72) Inventor: Limaye, Ajit M. 


AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


Hillsboro, Oregon 97124 (US) 


MC NL PT SE TR 




Designated Extension States: 


(74) Representative: Molyneaux, Martyn William 


AL LT LV MK RO SI 


Wildman, Harrold, Allen & Dixon 




11th Floor, Tower 3, 


(30) Priority: 06.11.2000 US 707520 


Clements Inn, 




London WC2A 2AZ (GB) 


(71) Applicant: Tektronix, Inc. 




Beaverton, OR 97077 (US) 





(54) Subliminally embedded keys in video for synchronization 



(57) A data file containing metadata and/or control 
data is synchronized with associated audio/video data 
by subliminally embedding a key in a frame of the audio/ 
video. The key indicates the location of the data file and 
a time after the embedded frame in which the data file 
is to be activated to be in synchronism with the audio/ 



video. The data file is accessed via an alternate channel 
and held in a buffer until the time indicated by the key. 
The metadata from the data file is played out for display 
in synchronism with the associated audio/video data, 
and instructions contained in the control data are exe- 
cuted in synchronism with the associated audio/video 
data. 
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Description 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to multimedia 
systems, and more particularly to a subliminally embed- 
ded key in a video signal tor synchronization of playout 
control of multimedia data and/or of execution of instruc- 
tions contained in control data. 
[0002] Watermarking of video signals is becoming of 
increasing use as a means of transmitting very low rate 
data as part of the video signal. Watermarking of video 
has been proposed for performing audio to video delay 
compensation, for indicating authentication or program 
ownership and for verifying playing of a video signal 
such as in advertisement logging. The watermarking 
technique inserts the very low rate data into the active 
portion of the video signal so as to be invisible while still 
having the robustness to survive video signal process- 
ing including multiple generations of compression en- 
coding/decoding. See U.S. Patent No. 4,969,041 issued 
to William O'Grady et al on November 6, 1990 entitled 
"Embedment of Data in a Video Signal" and co-pending 
U.S. Patent Application Serial No. 08/829,524 by Daniel 
Baker et al filed on March 28, 1 997 entitled "Transparent 
Embedment of Data in a Video Signal". 
[0003] Also the use of auxiliary channels fortransmit- 
ting data related to video signals is known, as shown in 
U.S. Patent No. 6,075,561 issued to Bozidar Janko on 
June 13, 2000 entitled "Low Duty-Cycle Transport of 
Video Reference Images". With the advent of the Inter- 
net, proposals have been made to use the Internet as 
the auxiliary channel for transporting data. It is anticipat- 
ed that when video that is streamed over the Internet is 
played out, it may be desirable to also play out other 
data — called metadata — that is associated with the 
video to immerse the viewer in the experience. It may 
also be desirable to transmit control data to the equip- 
ment receiving the video. This control data may be used 
for one or more of the following purposes: (1 ) cue a com- 
mercial; (2) specify a destination to which the video is 
to be routed; (3) control auxiliary equipment to actuate 
other sensory cues such as activation of special lighting 
or sound effects; (4) control the parameters for com- 
pressing the video prior to storage; etc. 
[0004] However transmitting the metadata is not cur- 
rently possible using watermarking technology with vid- 
eo. The bit rate of the metadata, such as 9600 baud, is 
too high for the very low data rate channel afforded by 
watermarking. Transmission of control data may or may 
not be possible using watermarking technology directly, 
depending upon the quantity of data which varies with 
the application. 

[0005] The problem that arises is how to synchronize 
the metadata or other control data with the video when 
they are transmitted over different channels so that the 
metadata appears at the proper point with relationship 
to the video when seen by the viewer, or the control data 



is retrieved and used at the proper point with relationship 
to the video. 

[0006] What is desired is a means for synchronizing 
the metadata or control data with the associated video 
5 when the video is played out. 

BRIEF SUMMARY OF THE INVENTION 

[0007] Accordingly the present invention provides 

10 subliminally embedded keys in video that act as pointers 
and synchronizing signals for a data file containing 
metadata and/or control data so that the metadata is dis- 
played with the video and/or instructions contained in 
the control data are executed at the proper time. At the 

15 transmission end of a system a key is generated in syn- 
chronization with the video, the key being associated 
with corresponding data file. The key is embedded sub- 
liminally in the video at an appropriate point using wa- 
termarking techniques. At a receiver the key is decoded 

20 from the video and used to access a network server that 
contains the data file. Once retrieved the metadata from 
the data file is displayed with the video at a point in syn- 
chronization with the video as indicated by the key, or 
the control data is used to perform one or more auxiliary 

25 functions at designated times. For small data files where 
the retrieval delay is small, the key may be just a few 
frames before the first video frame which is associated 
with the data file, while for large data files the key may 
be inserted several seconds before the first video frame 

30 which is associated with the data file. The key provides 
both an "address" for retrieving the data file and a time 
from the key frame when the metadata from the data file 
is to be played out with the video for a viewer or when 
the instructions contained in the control data are to be 

35 executed. In this way the data file is synchronized with 
the video. 

[0008] The objects, advantages and other novel fea- 
tures of the present invention are apparent from the fol- 
lowing detailed description when read in conjunction 
40 with the appended claims and attached drawing. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF 
THE DRAWING 

45 [0009] The Figure is a block diagram view of a system 
for synchronizing a data file with video using subliminally 
embedded keys according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

50 

[0010] Referring now to the Figure video, audio and 
an associated data file containing metadata and/or con- 
trol data is input to an edit/production system 12. The 
output from the edit/production system 12 is streamed 
55 audio/video and the data file with associated keys. The 
data file is stored in an appropriate storage device 14, 
such as a network server, and the keys are input with 
the streaming audio/video to a key embedment system 
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16 such as those described in U.S. Patent No. 
4,969,041 and U.S. Patent Application Serial No. 
08/829,524 referenced above. The keys have informa- 
tion that indicates the location of the associated data file 
in the storage medium 14 and a time in number of 5 
frames from the frame in which the key is embedded 
when the control instructions from the data file should 
be executed or the metadata should be played out to be 
in synchronism with the audio/video stream to account 
for latency in accessing the data file. Large data files 10 
require a longer access time, so the key is embedded 
in a frame much further ahead of the frame where the 
data file is to be used, whereas small data files may not 
require as much "lead" time. 

[0011] The streaming audio/video with embedded is 
keys is transmitted over a transmit channel to a receive 
site where a key decoder 18 extracts the keys from the 
streaming audio/video and accesses an alternate chan- 
nel 20, such as the Internet. The information from the 
keys is transmitted over the alternate channel 20 to the 20 
storage medium 14 and the data file is transferred from 
the storage medium to a buffer in the key decoder 18. 
When the number of frames from the frame in which the 
key is embedded has passed, as indicated by the key, 
the metadata from the data file is played out with the 25 
streaming audio/video to a playout system 22 so as to 
be synchronized with the audio/video stream. Alterna- 
tively the instructions contained in the control data of the 
data file are executed. The playout system 22 may be 
a set top box for a consumer television set or multimedia 30 
computer where the viewer can control whether or not 
to view the metadata or only portions of the metadata. 
The playout system 22 may also be the device respon- 
sible for implementing the instructions contained in the 
control data. 35 
[0012] Thus the present invention provides sublimi- 
nally embedded keys in video for synchronization of a 
data file containing metadata and/or control data with 
associated audio/video streams, the data file being pro- 
vided by an alternate channel and the keys having an 40 
address for the data file as well as a time when the meta- 
data in the data file plays out with the audio/video 
streams or when the instructions in the control data are 
executed. 

45 

Claims 



accessing the network storage medium accord- 
ing to the location indicated by the key; 
using the data file in synchronism with the au- 
dio/video data according to the latency time. 

The method as recited in claim 1 wherein the data 
file comprises metadata to be played out in synchro- 
nism with the associated audio/video data and the 
using step comprises the step of playing out the 
metadata for display in synchronism with the audio/ 
video data. 

The method as recited in claim 1 wherein the data 
file comprises control data to be activated in syn- 
chronism with the associated audio/video data and 
the using step comprises the step of activating in- 
structions contained in the control data for execu- 
tion in synchronism with the audio/video data. 



1. A method of synchronizing a data file with associ- 
ated audio/video data using subliminaify embedded 
keys comprising the steps of: 



50 



at a transmit site embedding a key subliminaify 
in the audio/video data, the key indicating a lo- 
cation of the data file on a network storage me- 
dium and a latency time; 
at a receive site extracting the key from the au- 
dio/video data; 
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